With the resource manager enabled for production usage, users should now be
able to run jobs. To demonstrate this, we will add a ``test'' user on the {\em master}
host that can be used to run an example job.

% begin_ohpc_run
\begin{lstlisting}[language=bash,keywords={}]
[sms](*\#*) useradd -m test
\end{lstlisting}
% end_ohpc_run

Next, the user's credentials need to be distributed across the cluster.
\xCAT{}'s \texttt{xdcp} has a merge functionality that adds new entries into
credential files on {\em compute} nodes: 

% begin_ohpc_run
\begin{lstlisting}[language=bash,keywords={}]
# Create a sync file for pushing user credentials to the nodes
[sms](*\#*) echo "MERGE:" > syncusers
[sms](*\#*) echo "/etc/passwd -> /etc/passwd" >> syncusers
[sms](*\#*) echo "/etc/group -> /etc/group"       >> syncusers
[sms](*\#*) echo "/etc/shadow -> /etc/shadow" >> syncusers
# Use xCAT to distribute credentials to nodes
[sms](*\#*) xdcp compute -F syncusers
\end{lstlisting}
% end_ohpc_run

\nottoggle{isxCATstateful}{Alternatively, the \texttt{updatenode compute -f} command
can be used. This re-synchronizes (i.e. copies) all the files defined in the
\texttt{syncfile} setup in Section \ref{sec:file_import}. \\ }  

~\\
\input{common/prun}

\iftoggle{isSLES_ww_slurm_x86}{\clearpage}
%\iftoggle{isxCAT}{\clearpage}

\subsection{Interactive execution}
To use the newly created ``test'' account to compile and execute the
application {\em interactively} through the resource manager, execute the
following (note the use of \texttt{prun} for parallel job launch which summarizes
the underlying native job launch mechanism being used):

\begin{lstlisting}[language=bash,keywords={}]
# Switch to "test" user
[sms](*\#*) su - test

# Compile MPI "hello world" example
[test@sms ~]$ mpicc -O3 /opt/ohpc/pub/examples/mpi/hello.c

# Submit interactive job request and use prun to launch executable
[test@sms ~]$ srun -n 8 -N 2 --pty /bin/bash

[test@c1 ~]$ prun ./a.out

[prun] Master compute host = c1
[prun] Resource manager = slurm
[prun] Launch cmd = mpiexec.hydra -bootstrap slurm ./a.out

 Hello, world (8 procs total)
    --> Process #   0 of   8 is alive. -> c1
    --> Process #   4 of   8 is alive. -> c2
    --> Process #   1 of   8 is alive. -> c1
    --> Process #   5 of   8 is alive. -> c2
    --> Process #   2 of   8 is alive. -> c1
    --> Process #   6 of   8 is alive. -> c2
    --> Process #   3 of   8 is alive. -> c1
    --> Process #   7 of   8 is alive. -> c2
\end{lstlisting}

\begin{center}
\begin{tcolorbox}[]
The following table provides approximate command equivalences between SLURM and
PBS Pro:

\vspace*{0.15cm}
\input common/rms_equivalence_table
\end{tcolorbox}
\end{center}
\nottoggle{isCentOS}{\clearpage}

\iftoggle{isCentOS}{\clearpage}

\subsection{Batch execution}

For batch execution, \OHPC{} provides a simple job script for reference (also
housed in the \path{/opt/ohpc/pub/examples} directory. This example script can
be used as a starting point for submitting batch jobs to the resource manager
and the example below illustrates use of the script to submit a batch job for
execution using the same executable referenced in the previous interactive example.

\begin{lstlisting}[language=bash,keywords={}]
# Copy example job script
[test@sms ~]$ cp /opt/ohpc/pub/examples/slurm/job.mpi .

# Examine contents (and edit to set desired job sizing characteristics)
[test@sms ~]$ cat job.mpi
#!/bin/bash

#SBATCH -J test               # Job name
#SBATCH -o job.%j.out         # Name of stdout output file (%j expands to %jobId)
#SBATCH -N 2                  # Total number of nodes requested
#SBATCH -n 16                 # Total number of mpi tasks #requested
#SBATCH -t 01:30:00           # Run time (hh:mm:ss) - 1.5 hours

# Launch MPI-based executable

prun ./a.out

# Submit job for batch execution
[test@sms ~]$ sbatch job.mpi
Submitted batch job 339
\end{lstlisting}

\begin{center}
\begin{tcolorbox}[]
\small
The use of the \texttt{\%j} option in the example batch job script shown is a convenient
way to track application output on an individual job basis. The \texttt{\%j} token
is replaced with the \SLURM{} job allocation number once assigned (job~\#339 in
this example).
\end{tcolorbox}
\end{center}


